{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/peremartra/Large-Language-Model-Notebooks-Course/blob/main/4-Evaluating%20LLMs/4_2_tracing_medical_agent.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f3cvnSVemTSU"
   },
   "source": [
    "<div>\n",
    "    <h1>Large Language Models Projects</a></h1>\n",
    "    <h3>Apply and Implement Strategies for Large Language Models</h3>\n",
    "    <h2>4.2-Tracing and Evaluating LLMs with LangSmith.\n",
    "  </h2>\n",
    "    <h3>Tracing a Medical Agent with LangSmith</h3>\n",
    "</div>\n",
    "\n",
    "by [Pere Martra](https://www.linkedin.com/in/pere-martra/)\n",
    "______________\n",
    "Models: OpenAI\n",
    "\n",
    "Colab Environment: CPU-High RAM.\n",
    "\n",
    "Keys:\n",
    "* Tracing LLMs\n",
    "* LangSmith\n",
    "* Agents.\n",
    "______________\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "knN-MUbEIMI8"
   },
   "source": [
    "In this notebook, we will explore how to trace the different calls that occur in a LangChain Agent using LangSmith. We will use a familiar agent, employed in the LangChain section of the course, where a RAG system with medical information was created.\n",
    "\n",
    "So, not only will we observe the traces of the agent, but we will also examine the traces of the retriever. Additionally, we'll inspect the query sent to the vectorial database and the returned results.\n",
    "__________"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KNW-LniD2I2o"
   },
   "source": [
    "#Installing libraries & Loading Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "pva9ehKXUpU2",
    "outputId": "7f8000eb-76f4-4e18-9ac3-1102362091c6"
   },
   "outputs": [],
   "source": [
    "!pip install -q langchain==0.3.0\n",
    "!pip install -q langchain-openai==0.2.0\n",
    "!pip install -q langchainhub==0.1.21\n",
    "!pip install -q datasets==3.0.0\n",
    "!pip install -q chromadb==0.5.5\n",
    "!pip install -q langchain-community==0.3.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KEwdHiGFkK_S"
   },
   "source": [
    "We will download the dataset from the Hugging Face datasets library. It's a dataset with information about diseases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "laSDMjqQXuj-"
   },
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "data = load_dataset(\"keivalya/MedQuad-MedicalQnADataset\", split='train')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 363
    },
    "id": "JnWZTcJiXzor",
    "outputId": "36b061d7-8cd9-4db9-8067-a45d88a66a75"
   },
   "outputs": [],
   "source": [
    "data = data.to_pandas()\n",
    "data.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "hf7RQa6B5xpx"
   },
   "outputs": [],
   "source": [
    "#uncoment this line if you want to limit the size of the data.\n",
    "data = data[0:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4FhUslovtiqn"
   },
   "source": [
    "As you can see, the medical information in the dataset is well-organized, and to someone like me, who is not an expert in the field, it appears to be quite valuable. This information could be a useful addition to any general medicine book to support primary care doctors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2sLkrHF6lWhM"
   },
   "source": [
    "Load the langchain libraries to load the document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "cCBAlIb596wZ"
   },
   "outputs": [],
   "source": [
    "from langchain.document_loaders import DataFrameLoader\n",
    "from langchain.vectorstores import Chroma"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nGCkVX6xldOR"
   },
   "source": [
    "The Document is in the Answer column, and the others columns are Metadata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "JZX8SaTe99Uf"
   },
   "outputs": [],
   "source": [
    "df_loader = DataFrameLoader(data, page_content_column=\"Answer\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 451
    },
    "id": "HDWnTqRY-IDr",
    "outputId": "b71a6cdd-9d34-46fd-f1ea-d138ccdd9387"
   },
   "outputs": [],
   "source": [
    "df_document = df_loader.load()\n",
    "display(df_document[:2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zPuTbSsXl3uF"
   },
   "source": [
    "We can chunk the documents. The size to which we want to split the document is a design decision. The larger it is, the larger the prompt will be, and the slower the Model's response process.\n",
    "\n",
    "We also need to consider the maximum prompt size and ensure that the document does not exceed it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8wogWV1_-lxh"
   },
   "outputs": [],
   "source": [
    "from langchain.text_splitter import CharacterTextSplitter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "kg7SRkqO-f9x"
   },
   "outputs": [],
   "source": [
    "text_splitter = CharacterTextSplitter(chunk_size=1250,\n",
    "                                      separator=\"\\n\",\n",
    "                                      chunk_overlap=100)\n",
    "texts = text_splitter.split_documents(df_document)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "y0dn9EfrKoeW"
   },
   "source": [
    "These warnings we see are because it can't perform the partition of the required size. This is because it waits for a page break to divide the text and does so when possible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "XaLWV_TzASSB",
    "outputId": "51ec4976-8e4f-4327-8b7f-9c416f43cb7e"
   },
   "outputs": [],
   "source": [
    "first_doc = texts[1]\n",
    "print(first_doc.page_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "B2_Pt7N6Zg2X"
   },
   "source": [
    "### Initialize the Embedding Model and Vector DB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "F_Dn06xGwjKP",
    "outputId": "31bffd76-ccfc-42a1-9926-676e81b8754c"
   },
   "outputs": [],
   "source": [
    "from getpass import getpass\n",
    "import os\n",
    "if not 'OPENAI_API_KEY' in os.environ:\n",
    "  os.environ[\"OPENAI_API_KEY\"] = getpass(\"OpenAI API Key: \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9NmsRgJ8rPoA"
   },
   "source": [
    "Obtain Your LangChain API Key from your Personal->Settings Area in LangSmith panel.\n",
    "\n",
    "![My Image](https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/img/langsmith_API_KEY.jpg?raw=true)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "xjuVU5rIrSRl",
    "outputId": "70352923-15bf-4db3-d2a9-6020c0be55de"
   },
   "outputs": [],
   "source": [
    "if not 'LANGCHAIN_API_KEY' in os.environ:\n",
    "  os.environ[\"LANGCHAIN_API_KEY\"] = getpass(\"LangChain API Key: \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jjZjm0JmrVsY"
   },
   "outputs": [],
   "source": [
    "os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n",
    "os.environ[\"LANGCHAIN_ENDPOINT\"]=\"https://api.smith.langchain.com\"\n",
    "os.environ[\"LANGCHAIN_PROJECT\"]=\"langsmith_test3\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "n8ROb8oMnRLD"
   },
   "source": [
    "We load the text-embedding-ada-002 model from OpenAI.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "U57x2_87YSpb"
   },
   "outputs": [],
   "source": [
    "from langchain_openai import OpenAIEmbeddings\n",
    "\n",
    "model_name = 'text-embedding-ada-002'\n",
    "\n",
    "embed = OpenAIEmbeddings(\n",
    "    model=model_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cgTCwF7UMyNW"
   },
   "source": [
    "The execution of this cell may take 3 to 5 minutes. If you want it to be faster, you can reduce the number of records in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "SEhQMQ8eCMj8"
   },
   "outputs": [],
   "source": [
    "directory_cdb = '/content/drive/MyDrive/chromadb'\n",
    "chroma_db = Chroma.from_documents(\n",
    "    df_document, embed, persist_directory=directory_cdb\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KhjV-T8GoarF"
   },
   "source": [
    "We are going to create three objects.\n",
    "\n",
    "* The language model, which can be any of those from OpenAI, the most common being gpt-3.5.\n",
    "* The memory, responsible for keeping the prompt with all the necessary history.\n",
    "* The retrieval, used to obtain information stored in ChromaDB."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "VksVhCjUnf8b"
   },
   "outputs": [],
   "source": [
    "model=\"gpt-4o\"\n",
    "#model=\"gpt-3.5-turbo\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "zMRs9Klic5-Y"
   },
   "outputs": [],
   "source": [
    "from langchain.chat_models import ChatOpenAI\n",
    "from langchain_openai import OpenAI\n",
    "from langchain.chains.conversation.memory import ConversationBufferWindowMemory\n",
    "from langchain.chains import RetrievalQA\n",
    "\n",
    "llm=OpenAI(temperature=0.0)\n",
    "\n",
    "conversational_memory = ConversationBufferWindowMemory(\n",
    "    memory_key='chat_history',\n",
    "    k=4, #Number of messages stored in memory\n",
    "    return_messages=True #Must return the messages in the response.\n",
    ")\n",
    "\n",
    "qa = RetrievalQA.from_chain_type(\n",
    "    llm=llm,\n",
    "    #chain_type=\"stuff\",\n",
    "    retriever=chroma_db.as_retriever()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ergrieE4o8qu"
   },
   "source": [
    "We can try the isolated Retrieval to see if the information it returns is relevant.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 107
    },
    "id": "LaYSq0V-dxHw",
    "outputId": "4a66e515-6323-4906-9410-405fe8e1cf9a"
   },
   "outputs": [],
   "source": [
    "qa.run(\"What is the main symptom of LCM?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pf9MXPeipEBO"
   },
   "source": [
    "When observing the Retriever on Langsmith is possible to see the input and the documents returned by it:\n",
    "![My Image](https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/img/Martra_Figure_4_Retriever_1.jpg?raw=true)\n",
    "\n",
    "In the image below you can observe the call to the Model where the full prompt and the response are displayed.\n",
    "![My Image](https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/img/Martra_Figure_4_Retriever_2.jpg?raw=true)\n",
    "\n",
    "\n",
    "## Creating the Agent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "FwCYrS4duqBW"
   },
   "outputs": [],
   "source": [
    "from langchain.agents import Tool\n",
    "\n",
    "#Defining the list of tool objects to be used by LangChain.\n",
    "tools = [\n",
    "    Tool(\n",
    "        name='Medical KB',\n",
    "        func=qa.run,\n",
    "        description=(\n",
    "            \"\"\"use this tool when answering medical knowledge queries to get\n",
    "            more information about the topic\"\"\"\n",
    "        )\n",
    "    )\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "JaKTzPUEvOoy",
    "outputId": "f2fceb76-32bd-4f2d-b562-ee70bded7034"
   },
   "outputs": [],
   "source": [
    "from langchain.agents import create_react_agent\n",
    "from langchain import hub\n",
    "\n",
    "prompt = hub.pull(\"hwchase17/react-chat\")\n",
    "agent = create_react_agent(\n",
    "    tools=tools,\n",
    "    llm=llm,\n",
    "    prompt=prompt,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "5sjTL1EU1vNW"
   },
   "outputs": [],
   "source": [
    "# Create an agent executor by passing in the agent and tools\n",
    "from langchain.agents import AgentExecutor\n",
    "agent_executor2 = AgentExecutor(agent=agent,\n",
    "                               tools=tools,\n",
    "                               verbose=True,\n",
    "                               memory=conversational_memory,\n",
    "                               max_iterations=30,\n",
    "                               max_execution_time=600,\n",
    "                               handle_parsing_errors=True\n",
    "                               )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IlxUBWKcvzeP"
   },
   "source": [
    "### Using the Conversational Agent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZZapCP4Pv2kz"
   },
   "source": [
    "To make queries we simply call the `agent` directly.\n",
    "\n",
    "First i will try a request not related to the Medical field."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "85vipqC02deV",
    "outputId": "c1383dcd-28be-4a1a-def4-eead285d44a6"
   },
   "outputs": [],
   "source": [
    "agent_executor2.invoke({\"input\": \"What is 2 multiplied by 2?\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LaTqyvBgG_wr"
   },
   "source": [
    "The initial call to the Agent happens with just one call to OpenAI. One piece of information available in LangSmith is the entire prompt. I'll copy it just below the image, so you can see.\n",
    "\n",
    "![My Image](https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/img/Martra_Figure_4_1AE_1.jpg?raw=true)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7LpYqjJuFKlw"
   },
   "source": [
    "\n",
    "```\n",
    "Assistant is a large language model trained by OpenAI.\n",
    "\n",
    "Assistant is designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, Assistant is able to generate human-like text based on the input it receives, allowing it to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand.\n",
    "\n",
    "Assistant is constantly learning and improving, and its capabilities are constantly evolving. It is able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. Additionally, Assistant is able to generate its own text based on the input it receives, allowing it to engage in discussions and provide explanations and descriptions on a wide range of topics.\n",
    "\n",
    "Overall, Assistant is a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether you need help with a specific question or just want to have a conversation about a particular topic, Assistant is here to assist.\n",
    "\n",
    "TOOLS:\n",
    "------\n",
    "\n",
    "Assistant has access to the following tools:\n",
    "\n",
    "Medical KB: use this tool when answering medical knowledge queries to get\n",
    "            more information about the topic\n",
    "\n",
    "To use a tool, please use the following format:\n",
    "\n",
    "\n",
    "Thought: Do I need to use a tool? Yes\n",
    "Action: the action to take, should be one of [Medical KB]\n",
    "Action Input: the input to the action\n",
    "Observation: the result of the action\n",
    "\n",
    "\n",
    "When you have a response to say to the Human, or if you do not need to use a tool, you MUST use the format:\n",
    "\n",
    "\n",
    "Thought: Do I need to use a tool? No\n",
    "Final Answer: [your response here]\n",
    "\n",
    "\n",
    "Begin!\n",
    "\n",
    "Previous conversation history:\n",
    "[]\n",
    "\n",
    "New input: What is 2 multiplied by 2?\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8suStMR7G11e"
   },
   "source": [
    "Perfect, the model has responded without accessing the configured knowledge database.\n",
    "\n",
    "Now I will try with a question that is also not related to health."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Wtwgfuy158LV"
   },
   "outputs": [],
   "source": [
    "agent_executor2.memory.clear()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "RJoAhy76vzAB",
    "outputId": "d2927d47-4f97-44dc-c7c3-dd05b220c1dc"
   },
   "outputs": [],
   "source": [
    "agent_executor2.invoke({\"input\": \"\"\"I have a patient that can have Botulism,\n",
    "how can I confirm the diagnosis?\"\"\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8lsaoF8nJNnR"
   },
   "source": [
    "Perfect, the most important thing for us is that it has been able to identify that it should go to the medical database to search for information about the symptoms.\n",
    "![My Image](https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/img/Martra_Figure_4_1AE_3.jpg?raw=true)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "mQeicHTj2pmY",
    "outputId": "092a21c6-53d1-4c06-9cb0-9820c6198dbe"
   },
   "outputs": [],
   "source": [
    "agent_executor2.invoke({\"input\": \"Is this an important illness?\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_uNIh-zXKefJ"
   },
   "source": [
    "On the left side of the image, you can see the various calls made by the Agent, which took 2.5 seconds to execute and consumed 2000 tokens.\n",
    "\n",
    "I will try to describe what happens in each call.\n",
    "\n",
    "**OpenAI**: he complete prompt, including the template from Hugging Face, and the user's question is passed to the OpenAI Model. It responds with the following step. Its answer is:\n",
    "\n",
    "*Thought: Do I need to use a tool?*\n",
    "\n",
    "*Yes  Action: Medical KB*  \n",
    "\n",
    "*Action Input: Botulism*\n",
    "\n",
    "**Medical KB**: The Agent utilizes the configured tool, passing only a single word as a parameter: \"Botulism.\"\n",
    "\n",
    "**Medical KB.Retriever:** The retriever returns four documents extracted from the Vectorial Database.\n",
    "\n",
    "**Medical KB.OpenAI:** The prompt is constructed with the information from the vectorial database. Even though I won't include the paste, I manage to identify that the four returned documents are actually two but duplicated. This, in a real project, could have helped me detect some issue, perhaps I have duplicates in the dataset, or maybe I loaded them twice. In any case, I'm consuming many more tokens than necessary. The model returns a response created considering the information contained in the prompt.\n",
    "\n",
    "**OpenAI:** In this final call to the model, it decides that the received response is what the user needs. Therefore, it marks it as correct and returns it to the user.\n",
    "\n",
    "![My Image](https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/img/Martra_Figure_4_1AE_3.jpg?raw=true)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gdyhyd6nJnwN"
   },
   "source": [
    "And the memory works perfectly. We can maintain a conversation, taking into account that the model knows the previous questions and answers.\n",
    "\n",
    "\n",
    "# Conclusions.\n",
    "LangSmith is an incredibly useful tool for tracing and storing all the information generated when making calls to LangChain.\n",
    "\n",
    "The experiment has been a small success. The Vectorial database has been configured and filled with information from the dataset. A LangChain agent has been created, and it has been able to retrieve information from the database only when necessary. Don't forget that our ChatBot has memory.\n",
    "\n",
    "\n",
    "\n",
    "And you have all the information in a project stored in LangSmith!!!!\n",
    "![My Image](https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/img/Martra_Figure_4_1AE_Final.jpg?raw=true)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ykg5TYA033yR"
   },
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "gpuType": "V100",
   "include_colab_link": true,
   "machine_shape": "hm",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
